Causal Reinforcement Learning: An Instrumental Variable Approach

In the standard data analysis framework, is first collected (once for all), and then carried out. Moreover, data-generating process typically assumed to be exogenous. This approach natural when analyst has no impact on how generated. The advancement of digital technology, however, facilitated firms learn from make decisions at same time. As these generate new data, analyst---a business manager or an algorithm---also becomes generator. this article, we formulate problem as a Markov decision (MDP) show that interaction generates type bias---reinforcement bias---that exacerbates endogeneity in static analysis. When are independent identically distributed, embed instrumental variable (IV) stochastic gradient descent algorithm correct bias. For general MDP problems, propose class IV-based reinforcement learning (RL) algorithms We establish asymptotic properties by incorporating them into two-timescale approximation (SA). Our formulation requires unbounded state space more importantly, Markovian noise. Therefore, techniques RL SA literature, which rely boundedness martingale-difference structure noise, do not apply. develop finite-time risk bounds, bounds trajectory stability, distribution IV-RL algorithms.